NEW DIRECTIONS IN NIELSEN-REIDEMEISTER 

THEORY 



ALEXANDER FEL'SHTYN 

Abstract. The purpose of this expository paper is to present new 
directions in the classical Nielsen-Reidemeister fixed point theory. 
We describe twisted Burnside-Frobenius theorem, groups with 
property and a connection between Nielsen fixed point theory and 
symplectic Floer homology. 
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1. Introduction 

1.1. Nielsen- Reidemeister fixed point theory. Let f : X X 

be a map of a compact topological space X. Nielsen- Reidemeister 
fixed point theory suggests a way of counting the fixed points of the 
map / in the presence of the fundamental group of the space X. Let 
p : X ^ X he the universal covering of X and / : X — > X a lifting 
of /, ie. p o f = f o P- Two liftings / and /' are called conjugate 
if there is a 7 G F = vri(X) such that /' = 7 o / o 7"^. The subset 
p{Fix{f)) C Fix{f) is called the fixed point class of f determined 
by the lifting class [/]. A fixed point class is called essential if its 
index is nonzero. The number of lifting classes of / (and hence the 
number of fixed point classes, empty or not) is called the Reidemeister 
Number of f, denoted R{f)- This is a positive integer or infinity. The 
number of essential fixed point classes is called the Nielsen number 
of /, denoted by N{f). The Nielsen number is always finite. R{f) 
and A^(/) are homotopy type invariants. In the category of compact, 
connected polyhedra the Nielsen number of a map is equal to the least 
number of fixed points of maps with the same homotopy type as /. In 
Nielsen fixed point theory the main objects for investigation are the 
Nielsen and Reidemeister numbers and their modifications [58]. 

Our definition of a fixed point class is via the universal covering 
space. It essentially says: Two fixed point of / are in the same class 
iff there is a lifting / of / having fixed points above both of them. 
There is another way of saying this, which does not use covering space 
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explicitly, hence is very useful in identifying fixed point classes. Namely, 
two fixed points xq and xi of / belong to the same fixed point class 
iff there is a path c from xq to Xi such that c = / o c ( homotopy 
relative endpoints). This can be considered as an equivalent definition 
of a non-empty fixed point class. Every map / has only finitely many 
non-empty fixed point classes, each a compact subset of X. Given a 
homotopy H = {ht} : fo = fi, we want to see its influence on fixed 
point classes of /o and /i. A homotopy H = {ht} : X X is called 
a lifting of the homotopy H = {ht}, ii ht is a. lifting of ht for every 
t E I. Given a homotopy H and a lifting /q of /o, there is a unique 
lifting H of H such that ho = /o, hence by unique lifting property of 
covering spaces they determine a lifting /i of /i. Thus H gives rise 
to a one-one correspondence from liftings of /o to liftings of /i. This 
correspondence preserves the conjugacy relation. Thus there is a one- 
to-one correspondence between lifting classes and fixed point classes of 
/o and those of fi. 

Given a selfmap / : X — > X of a compact connected manifold X, the 
nonvanishing of the classical Lefschetz number L{f ) guarantees the ex- 
istence of fixed points. Unfortunately, L{f) yields no information about 
the size of the set of fixed points of /. However, the Nielsen number 
N{f), a more subtle homotopy invariant, provides a lower bound on 
the size of this set. For dimX > 3, a classical theorem of Wecken 
asserts that A^(/) is a sharp lower bound on the size of this set, that is, 
A^(/) is the minimal number of fixed points among all maps homotopic 
to /. Thus the computation of A^(/) is a central issue in fixed point 
theory. 

Let G be a countable discrete group and (p : G ^ G an endomor- 
phism. Two elements x,x' E G are said to be cf)- conjugate or twisted 
conjugate, iff there exists g E G with x' = gx(f){g~^). We shall write 
{x}^ for the (p-conjugacy or twisted conjugacy class of the element 
X E G. The number of 0-conjugacy classes is called the Reidemeis- 
ter number of an endomorphism and is denoted by R{4>). If (f) is 
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the identity map then the 0-conjugacy classes are the usual conjugacy 
classes in the group G. 

Let / : X — i> X be given, and let a specific lifting f : X ^ X he 
chosen as reference. Let F be the group of covering translations of 
X over X. Then every lifting of / can be written uniquely as a o /, 
with a G r. So elements of F serve as coordinates of liftings with 
respect to the reference /. Now for every a G F the composition 
/ o a is a lifting of / so there is a unique a' G F such that a' o f = 
f o a. This correspondence a ^ a' is determined by the reference /, 
and is obviously a homomorphism. The endomorphism : F F 
determined by the lifting / of / is defined by o / = / o a. It is 

well known that F = 7ri{X). We shall identify tt = iti{X,xq) and F in 
the usual way. 

We have seen that a G vr can be considered as the coordinate of 
the lifting a o f. We can tell the conjugacy of two liftings from their 
coordinates: [aof] = [a' of] iff there is 7 G vr such that a' = 7a/* (7""'^). 

So we have the Reidemeister bijection: Lifting classes of / are in 1-1 
correspondence with /^-conjugacy classes in group vr, the lifting class 
[a o /] corresponds to the /*-cojugacy class of a. 

By an abuse of language, we say that the fixed point class p(Fix(a o 
/)), which is labeled with the lifting class [a o /], corresponds to the 
/*-conjugacy class of a. Thus the /^.-conjugacy classes in vr serve as 
coordinates for the fixed point classes of /, once a reference lifting / is 
chosen. 

1.2. New directions. The interest in twisted conjugacy relation for 
group endomorphism has its origins not only in Nielsen-Reidemeister 
fixed point theory (see, e.g. [82, 58, 22]), but also in Selberg theory 
(see, eg. [90, 2]) and Algebraic Geometry (see, e.g. [50]). 

A current important problem in this area is to obtain a twisted ana- 
logue of the celebrated Burnside-Frobenius theorem [25, 22, 32, 33, 95, 
28, 31, 29, 54], that is, to show the equahty of the Reidemeister number 
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of and the number of fixed points of the induced homeomorphism of 
an appropriate dual object. 

If G is a finite group, then the classical Burnside-Frobenius theorem 
(see, e.g., [88], [63, p. 140]) says that the number of classes of irreducible 
representations is equal to the number of conjugacy classes of elements 
of G. Let G be the unitary dual of G, i.e. the set of equivalence classes 
of unitary irreducible representations of G. 

If : G ^ G is an automorphism, it induces a map (f) : G ^ G, 
(j){p) = p o (j). Therefore, by the Burnside-Frobenius theorem, if is 
the identity automorphism of any finite group G, then we have -R(0) = 
#Fix(0). 

In [25] it was discovered that this statement remains true for any 
automorphism of any finite group G. Indeed, if we consider an au- 
tomorphism of a finite group G, then -R(0) is equal to the dimension 
of the space of twisted invariant functions on this group. Hence, by 
Peter- Weyl theorem (which asserts the existence of a two-side equi- 
variant isomorphism G*(G) = 0^gg End(^rp)), -R(0) is identified with 
the sum of dimensions dp of twisted invariant elements of End(iirp), 
where p runs over G, and the space of representation p is denoted by 
Hp. By the Schur lemma, dp = 1, if p is a fixed point of 0, and is zero 
otherwise. Hence, -R(0) coincides with the number of fixed points of 0. 

The attempts to generalize this theorem to the case of non-identical 
automorphism and of non- finite group (i.e., to identify the Reidemeis- 
ter number of and the number of fixed points of on an appropriate 
dual object of G, provided that one of these numbers is finite) were 
inspired by the dynamical questions and were the subject of a series 
of papers [25, 22, 32, 33, 95, 30, 28, 31, 29, 54] In the paper [35] we 
studied the following property for a countable discrete group G and 
its automorphism 0: we say that the group is cf)- conjugacy separa- 
ble if its Reidemeister classes can be distinguished by homomorphisms 
onto finite groups, and we say that it is twisted conjugacy separable if 
it is 0-conjugacy separable for any automorphism with -R(0) < oo 
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{strongly twisted conjugacy separable, if we remove this finiteness re- 
striction) (Definitions 2.3 and 2.5). This notion was used in [31] to 
prove the twisted Burnside-Frobenius theorem for polycychc-by-finite 
groups with the finite-dimensional part of the unitary dual G as an 
appropriate dual object. 

In chapter 2, after some preliminary considerations, we discuss fol- 
lowing results: 

(1) Classes of twisted conjugacy separable groups: Polycyclic-by- 
finite groups are strongly twisted conjugacy separable groups 
[31, 35]. 

(2) Twisted conjugacy separability respects some extensions: Sup- 
pose, there is an extension H ^ G ^ G/H, where the group 
if is a characteristic twisted conjugacy separable group; G/H 
is finitely generated FC-group (i.e., a group with finite conju- 
gacy classes). Then G is a twisted conjugacy separable group 
[31, 35]. 

(3) Examples of groups, which are not twisted conjugacy separable: 
HNN, Ivanov and Osin groups [31]. 

(4) The affirmative answer to the twisted Dehn conjugacy problem 
for polycyclic- by-finite groups [35]. 

(5) Twisted Burnside-Frobenius theorem for 0-conjugacy separable 
groups in the following formulation: Let G be an 0-conjugacy 
separable group .Then R{(f)) = Sf{(j)) if one of these numbers is 
finite, where Sf{(j)) is the number of fixed points of </> : G/ ^ Gf, 
(j){p) = p o 0, where Gj is the part of the unitary dual G, which 
is formed by the finite-dimensional representations [31, 35]. 

A number of examples of groups and automorphisms with finite Rei- 
demeister numbers was obtained and studied in [22, 44, 26, 33, 30]. Us- 
ing the same argument as in [32] one obtains from the twisted Burnside- 
Frobenius theorem the following dynamical and number-theoretical 
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consequence which, together with the twisted Burnside-Frobenius the- 
orem itself, is very important for the reahzation problem of Reidemeis- 
ter numbers in topological dynamics and the study of the Reidemeister 
zeta-f unction. Let fi{d), (i G N, be the Mohius function, i.e.fi{d) = if 
d is divisible by a square different from one ; fi{d) = (—1)^ if d is not 
divisible by a square different from one , where k denotes the number 
of prime divisors of = 1. 

Congruences for Reidemeister numbers[31]: Let (p : G ^ G be an 
automorphism of a countable discrete twisted conjugacy separable group 
G such that all numbers R{(j)"') are finite. Then one has for all n, 



These congruences were proved previously in a number of special 
cases in [25, 22, 32, 33, 30] and are an analog of remarkable Dold 
congruences for the Lefschetz numbers of the iterations of a continuous 
map [16]. 

One step in the process to obtain twisted Burnside-Frobenius theo- 
rem is to describe the class of groups G for which -R(0) = oo for any 
automorphism (p : G ^ G. We say that a group G has i?oo property if 
all of its automorphisms (p satisfy R{(f)) = oo. 

The work of discovering which groups have -Roo property was begun 
by Fel'shtyn and Hill in [25]. It was later shown by various authors that 
the following groups have Roo property : (1) non-elementary Gromov 
hyperbolic groups [23, 65], (2) Baumslag-Solitar groups BS{m,n) = 
{a,b\ba"^b~^ = a") except for 55(1,1) [24], (3) generalized Baumslag- 
Solitar groups, that is, finitely generated groups which act on a tree 
with all edge and vertex stabilizers infinite cyclic [64], (4) lamplighter 
groups Z„ I Z if and only if 2\n or 3\n [46], (5) the solvable generaliza- 
tion r of BS{l,n) given by the short exact sequence 1 — Z[i] ^ F ^ 
Z'^ ^ 1, as well as any group quasi-isometric to F [91]; such groups 
are quasi-isometric to ^^(l,^) [92] ( note however that the class of 




mod n. 
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groups for which R{(f)) = oo for any automorphism is not closed un- 
der quasi-isometry) (6) saturated weakly branch groups (including the 
Grigorchuk group and the Gupta-Sidki group) [34], (7)The R. Thomp- 
son group F [7], (8) symplectic groups Sp{2n,'Z), the mapping class 
groups Mods of a compact surface S and the full braid groups Bn{S) 
on n strings of a compact surface S in the cases where 5* is either the 
compact disk D, or the sphere 5*^ [36]. 

The results of the present paper indicate that the further study of 
Reidemeister theory for these groups should go along the lines similar 
to those of the infinite case. On the other hand, this result reduces the 
class of groups for which the twisted Burnside-Frobenius conjecture 
[25, 32, 33, 95, 31, 29, 54] has yet to be verified. 

One is interested in conditions which yield more specific informa- 
tion about the relationship between the Lefshetz number, the Nielsen 
number and the Reidemeister number. In the special case of selfmaps 
of a lens space, Franz [41] showed that all fixed point classes of such 
maps have the same fixed point index. From this it follows that one 
of two situations occurs, namely 1) L{f) = implies N{f) = 0, in 
which case / is deformable to a fixed point free map, and 2) L{f) ^ 
implies N{f) = ^Coker{l — f^^) where is the induced homomor- 
phism on the first integral homology. When the Lefschetz number 
is nonzero, the cardinality of Coker{l — /*J is exactly equal to 
the Reidemeister number R{f)- 

The evaluation subgroup J{X), also known as the first Gottlieb 
group Gi{X), of the fundamental group vr is the image of the homomor- 
phism ev^ : 7Ti{X^ , Ix) — > '7ri(X, Xq) induced by the evaluation map at 
a point xq G X. If J{X) coincides with vr, the analogous results are ob- 
tained as well. Namely, L{f) = implies that N{f) = 0, and L{f) ^ 
implies that A^(/) = -R(/) for all maps /. This general type of result 
was first proved by B. Jiang (see [58] for more information on Nielsen 
fixed point theory) for a class of spaces now known as Jiang spaces. 
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Jiang-type results hold for all selfmaps of a large class of spaces includ- 
ing simply-connected spaces, generalized lens spaces, if-spaces, topo- 
logical groups, orientable coset spaces of compact connected Lie groups, 
nilmanifolds, certain C-nilpotent spaces where C denotes the class of fi- 
nite groups, certain solvmanifolds and infra- homogeneous spaces [96]. 
When dimX > 3 and N{f) = 0, the selfmap / is deformable to be 
fixed point free by Wecken's theorem. For the equality N{f) = R{f) 
to hold, one must first determine the finiteness of the Reidemeister 
number R{f). 

Unless N{f) = for all homeomorphisms /, property R^o eliminates 
the possibility of a Jiang-type result, which would require the finiteness 
of the Reidemeister number. That is, if a group G has property R^o, a 
compact connected manifold with G as fundamental group will never 
satisfy a Jiang-type result. 

In particular the fundamental group of surface of genus greater than 
1 has property Roo- So, we need a new ideas and tools in this classical 
case. Recently a connection between symplectic Floer homology and 
Nielsen fixed point theory was discovered [37, 43]. The author came to 
the idea that Nielsen numbers are connected with Floer homology of 
surface diffeomorphisms at the Autumn 2000, after conversations with 
Joel Robbin and Dan Burghelea. 

In the chapter 4 we discuss the connection between symplectic Floer 
homology group and Nielsen fixed point theory. We also describe sym- 
plectic zeta functions and an asymptotic invariant of monotone sym- 
plectomorphism. Generalised Arnold conjecture is formulated. 

A part of this article grew out of the joint papers of the author 
with E.V. Troitsky and D. L. Gongalves written in 2001- 2007. Their 
collaboration is gratefully appreciated. 

The author is grateful to B. Bowditch, F. Dahmani, Ya. Eliashberg, 
M. Gromov, V. Guirardel, Wu-Chung Hsiang, M. Kapovich, G. Levitt, 
J.D. McCarthy, K. Ono, M. Sapir, Y. Rudyak, S. Sidki, V. Turaev and 
A. Vershik for stimulating discussions and comments. 
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Parts of this article were written while the author was was visiting 
the Max-Planck- Institute fiir Mathematik, Bonn in 2001-2006. The 
author would like to thank the Max-Planck-Institute fiir Mathematik, 
Bonn for kind hospitality and support. 

2. Twisted Burnside-Frobenius theorem 

2.1. Preliminary considerations. Following construction relates </>- 
conjugacy classes and some conjugacy classes of another group. Con- 
sider the action of Z on G, i.e. a homomorphism Z Aut(G), n i— > {/)"■. 
Let r be a corresponding semi-direct product P = G xi Z: 

(2.1) T ■=<G,t\tgt~^ = (pig) > 

in terms of generators and relations, where t is a generator of Z. The 
group G is a normal subgroup of P. As a set, P has the form 

(2.2) P = U^ezG ■ r, 
where G ■ is the coset by G containing t". 

Remark 2.1. Any usual conjugacy class of P is contained in some 
G ■ r. Indeed, gg't^'g"^ = gg'4>'^{g-^)r and tgTt-^ = 4>{g')t'^. 

Lemma 2.1. Two elements x,y of G are (p- conjugate iff xt andyt are 
conjugate in the usual sense in P. Therefore g ^ g-t is a bijection from 
the set of (p- conjugacy classes of G onto the set of conjugacy classes of 
P contained in G ■ t. 

Proof. If X and y are </)-conjugate then there is a (7 G G such that 
gx = y(f){g)- This implies gx = ytgt~^ and therefore g{xt) = {yt)g so 
xt and yt are conjugate in the usual sense in P. Conversely, suppose xt 
and yt are conjugate in P. Then there is a gt"- G P with gf^xt = ytgt"-. 
From the relation txt~^ = (f){x) we obtain g(f)"'{x)t"'~^^ = y(j){g)t"'~^^ and 
therefore g(f)"'{x) = y(f){g). Hence, y and 0"(x) are 0-conjugate. Thus, 
y and x are 0-conjugate, because x and 0(x) are always 0-conjugate: 

(f){x) = X~^X(f){x). □ 
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2.2. Twisted conjugacy separability. We would like to give a gen- 
eralization of the following well known notion. 

Definition 2.2. A group G is conjugacy separable if any pair g, h of 
non-conjugate elements of G are non-conjugate in some finite quotient 
of G. 

It was proved that polycyclic- by-finite groups are conjugacy sepa- 
rated ([83, 40], see also [85, Ch. 4]). 

We can introduce the following notion, which coincides with the 
previous definition in the case = Id. 

Definition 2.3. [35] A group G is (p- conjugacy separable with respect 
to an automorphism (p : G ^ G ii any pair g, h of non-0-conjugate ele- 
ments of G are non-0-conjugate in some finite quotient of G respecting 



This notion is closely related to the notion RP(0) introduced in [31]. 

Definition 2.4. [31] We say that a group G has the property RP if for 
any automorphism (p with R{4>) < oo the characteristic functions / of 
Reidemeister classes (hence all 0-central functions) are periodic in the 
following sense. 

There exists a finite group K, its automorphism (px, and epimor- 
phism F : G ^ K such that 

(1) The diagram 

G G 



F 



F 



<PK 

K ^ K 

commutes. 

(2) / = F*fK, where fx is a characteristic function of a subset of 
K. 

If this property holds for a concrete automorphism (p, we will denote 
this by RP(0). 
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One gets immediately the following statement. 

Theorem 2.2. [31] Suppose, R{4>) < oo. Then G is (j)-conjugacy sep- 
arable if and only if G is RP(0). 

Proof. Indeed, let Fij : G Kij distinguish ith and jth 0-conjugacy 
classes, where Kij are finite groups, i,j = 1, . . . ,R{(f)). Let F : G ^ 
(BijKij, F{g) = ^ijid)^ be the diagonal mapping and K its image. 
Then F : G ^ K gives RP(0). 

The opposite implication is evident. □ 

Definition 2.5. [35] A group G is twisted conjugacy separable if it is 
0-conjugacy separable for any with -R(0) < oo. 

A group G is strongly twisted conjugacy separable if it is 0-conjugacy 
separable for any (p. 

From Theorem 2.2 one immediately obtains 

Corollary 2.3. [35] A group G is twisted conjugacy separable if and 
only if it is RP. 

Theorem 2.4. [31] Let F : T K be a morphism onto a finite group 
K which separates two conjugacy classes of T in G ■ t. Then the re- 
striction Fg '■= F\g '■ G lm{F\G) separates the corresponding [by 
the bijection from Lemma 2.1) (p-conjugacy classes in G. 

Proof. First of all let us remark that Ker(FG') is 0-invariant. Indeed, 
suppose Fcig) = F{g) = e. Then 



(the kernel of F is a normal subgroup). 

Let gt and gt be some representatives of the mentioned conjugacy 
classes. Then 



fg{M) = HM) = Fitgt-i = mm 



-1 



= e 



F{{ht-)gt{ht-)-') ^ F{gt), 



yheG,neZ, 



F{ht-gt) ^ F{gtht-), 



yheG,neZ, 
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F{h(t)''{g)e^^) ^ F(g(P{h)e+^), ^heG^neZ, 
F{hr{g)) ^ F{gm), WheG,neZ, 
in particular, F{hg(f){h^^)) ^ F(g) Mh e G. □ 

Theorem 2.5. [35] Let some class of conjugacy separable groups be 
closed under taking semidirect products by Z. Then this class consists 
of strongly twisted conjugacy separable groups. 

Proof. This foUows immediately from Theorem 2.4 and Theorem 2.2. 

□ 

2.2.1. First examples: poly cyclic- by- finite groups. As an application 
we obtain another proof of the main theorem for polycyclic-by-finite 
groups. 

Let G' = [G, G] be the commutator subgroup or derived group of G, 
i.e. the subgroup generated by commutators. G' is invariant under any 
homomorphism, in particular it is normal. It is the smallest normal 
subgroup of G with an abelian factor group. Denoting G^^^ := G, 
G'(i) := G', G^") := n>2, one obtains derived senes of G: 

(2.3) G = dG' D D . . . D D ... 

If = e for some n, i.e. the series (2.3) stabilizes by trivial group, 
the group G is solvable; 

Definition 2.6. A solvable group with derived series with cyclic factors 
is called polycyclic group. 

Theorem 2.6. [31] Any polycyclic-by-finite group is a strongly twisted 
conjugacy separable group. 

Proof. The class of polycyclic-by-finite groups is closed under taking 
semidirect products by Z. Indeed, let G be an polycyclic-by-finite 
group. Then there exists a characteristic (polycyclic) subgroup P of 
finite index in G. Hence, P x Z is a polycyclic normal group of G x Z 
of the same finite index. 
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Polycyclic-by-finite groups are conjugacy separable ([83, 40], see also 
[85, Ch. 4]). It remains to apply Theorem 2.5. □ 

2.3. Twisted conjugacy separability and extensions. It is known 
that conjugacy separability does not respect extensions. For twisted 
conjugacy separable groups the situation is much better under some 
finiteness conditions. More precisely one has the following statement. 

Theorem 2.7. [31] Suppose, there exists a commutative diagram 
(2.4) H G — G/H 



H — G — G/H 0, 

where H is a normal subgroup of a finitely generated group G. Suppose, 
R{(/)) < oo, G/H is a FC group, i.e., all conjugacy classes are finite, 
and H is a (p' -conjugacy separable group. Then G is a (p-conjugacy 
separable group. 

Another variant of finiteness is \G/H\ < oo (without the property 
i?(0) < oo). 

2.4. Twisted Burnside-Probenius theorem for 0-conjugacy sep- 
arable groups. 

Definition 2.7. Denote by G/ the subset of the unitary dual G related 
to finite-dimensional representations. 

Theorem 2.8. [31] Let G be an (p-conjugacy separable group .Then 
R{(f)) = Sf{(f)) if one of these numbers is finite. 

Proof. The coefficients of finite-dimensional non-equivalent irreducible 
representations of G are linear independent by Frobenius-Schur the- 
orem (see [12, (27.13)]). Moreover, the coefficients of non-equivalent 
unitary finite-dimensional irreducible representations are orthogonal to 
each other as functions on the universal compact group associated with 
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the initial group [14, 16.1.3] by the Peter- Weyl theorem. Hence, their 
hnear combinations are orthogonal to each other as well. 
It is sufficient to verify the following three statements: 

1) If -R(0) < oo, then each 0-class function is a finite linear com- 
bination of twisted-invariant functionals being coefficients of points of 
Fix (j) f . 

2) If p G Fix(f)f, there exists one and only one (up to scaling) twisted 
invariant functional on p{C*{G)) (this is a finite full matrix algebra). 

3) For different p the corresponding 0-class functions are linearly 
independent. This follows from the remark at the beginning of the 
proof. Let us remark that the property RP implies in particular that 
0-central functions (for with R{(f)) < oo) are functionals on C*{G), 
not only L^{G), i.e. are in the Fourier-Stieltijes algebra B{G). 

The statement 1) follows from the RP property. Indeed, this (p- 
class function / is a linear combination of functionals coming from 
some finite collection {pi} of elements of Gf (these representations 
Pi, . . . , Ps are in fact representations of the form VTj o F, where VTj are 
irreducible representations of the finite group K and F : G ^ K, as in 
the definition of RP). So, 

/ = $^/.°P^, pr.G -^EndiVi), fr.End{Vi)^C,p,^pj,{z^j). 

i=l 

For any g^lj E G one has 

s s 

Y.f'M9)) = fig) = figMg-')) = Y.f^^P^^9mg-')))- 

i=l 1=1 

By the observation at the beginning of the proof concerning linear 
independence, 

Mpiig)) = fi{pi{gg<P{g~^)))- z = i, . . . , s, 

i.e. fi are twisted-invariant. For any p E Gj, p : G ^ End(V^), any 
functional u : End(y) — > C has the form a Tr(6a) for some fixed 
b G End(V^). Twisted invariance implies twisted invariance of b (evident 
details can be found in [32, Sect. 3]). Hence, b is intertwining between 
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p and p o (p and p G Fix(0j). The uniqueness of intertwining operator 
(up to scaling) implies 2). 

It remains to prove that -R(0) < oo if Sf{(f)) < oo. By the definition 
of a 0-conjugacy separable group the Reidemeister classes of can be 
separated by maps to finite groups. Hence, taking representations of 
these finite groups and applying the twisted Burnside-Frobenius theo- 
rem to these groups we obtain that for any pair of Reidemeister classes 
there exists a function being a coefficient of a finite-dimensional unitary 
representation, which distinguish these classes. Hence, if R{4>) = oo, 
then there are infinitely many linearly independent twisted invariant 
functions being coefficients of finite dimensional representations. But 
there are as many such functionals, as Sf{(j)). 

□ 

Corollary 2.9. [31] Let G be almost polycyclic group and its auto- 
morphism. Then R{(f)) = Sf{(j)) if one of these numbers is finite. 

2.5. Examples and counterexamples. Some of examples of groups, 
for which the twisted Burnside-Frobenius theorem in the above formu- 
lation is true, out of the class of polycyclic-by-finite groups were ob- 
tained by F. Indukaev [54]. Namely, it is proved that wreath products 
A I Jj are RP groups, where A is a finitely generated abelian group 
(these groups are residually finite). 

Now let us present some counterexamples to the twisted Burnside- 
Frobenius theorem in the above formulation for some discrete groups 
with extreme properties. Suppose, an infinite discrete group G has a 
finite number of conjugacy classes. Such examples can be found in [89] 
(HNN-group), [76, p. 471] (Ivanov group), and [77] (Osin group). Then 
evidently, the characteristic function of the unity element is not almost- 
periodic and the argument above is not valid. Moreover, let us show, 
that these groups give rise counterexamples to the above theorem. 

In particular, they are not twisted conjugacy separable. Evidently, 
they are not conjugacy separable, because they are not residually finite. 



NEW DIRECTIONS IN NIELSEN-REIDEMEISTER THEORY 17 

Example 2.8. [31] For the Osin group Reidemeister number R{ld) = 
2, while there is only trivial (1-dimensional) finite-dimensional repre- 
sentation. Indeed, Osin group is an infinite finitely generated group 
G with exactly two conjugacy classes. All nontrivial elements of this 
group G are conjugate. So, the group G is simple, i.e. G has no non- 
trivial normal subgroup. This implies that group G is not residually 
finite (by definition of residually finite group). Hence, it is not linear 
(by Mal'cev theorem [68], [84, 15.1.6]) and has no finite-dimensional 
irreducible unitary representations with trivial kernel. Hence, by sim- 
plicity of G, it has no finite-dimensional irreducible unitary represen- 
tation with nontrivial kernel, except of the trivial one. 

Let us remark that Osin group is non-amenable, contains the free 
group in two generators F2, and has exponential growth. 



Example 2.9. [31] For large enough prime numbers p, the first ex- 
amples of finitely generated infinite periodic groups with exactly p 
conjugacy classes were constructed by Ivanov as limits of hyperbolic 
groups (although hyperbolicity was not used explicitly) (see [76, The- 
orem 41.2]). Ivanov group G is infinite periodic 2-generator group, in 
contrast to the Osin group, which is torsion free. The Ivanov group G 
is also a simple group. The proof (kindly explained to us by M. Sapir) 
is the following. Denote by a and b the generators of G described in [76, 
Theorem 41.2]. In the proof of Theorem 41.2 on [76] it was shown that 
each of elements of G is conjugate in G to a power of generator a of 
order s. Let us consider any normal subgroup of G. Suppose j E N. 
Then 7 = ga^g~^ for some g E G and some s. Hence, a* = g~^'yg G 
and from periodicity of a, it follows that also a G A^ as well as a'^ G A^ 
for any k, because p is prime. Then any element h of G also belongs 
to A^ being of the form h = ha^{h)~^, for some /c, i.e., N = G. Thus, 
the group G is simple. The discussion can be completed in the same 
way as in the case of Osin group. 
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Example 2.10. In paper [51], Theorem III and its corollary, G. Hig- 
man, B. H. Neumann, and H. Neumann proved that any locally infinite 
countable group G can be embedded into a countable group G* in which 
all elements except the unit element are conjugate to each other (see 
also [89]). The discussion above related Osin group remains valid for 
G* groups. 

2.6. Twisted Dehn conjugacy problem. The subject is closely re- 
lated to some decision problem. Recall that M. Dehn in 1912 [13] (see 
[67, Ch. 1, §2; Ch. 2, §1])) has formulated in particular 
Conjugacy problem: Does there exists an algorithm to determine 
whether an arbitrary pair of group words U , V in the generators of G 
define conjugate elements of Gl 

Following question was posed by G. Makanin [71, Question 10.26(a)]: 

Question: Does there exists an algorithm to determine whether for 
an arbitrary pair of group words U and of a free group G and an 
arbitrary automorphism of G the equation (f){X)U = VX solvable in 
G? 

In [8] the affirmative answer to the Makanin's question is obtained. 

In [3] the following problem, which generalizes the two above prob- 
lems, was posed: 

Twisted conjugacy problem: Does there exists an algorithm to 
determine whether for an arbitrary pair of group words U and V in the 
generators of G the equality (j){X)U = VX holds for some W & G and 
(f) E H, where is a fixed subset of Aut(G)? 

We will discuss the twisted conjugacy problem for H = {(p}. 

Theorem 2.10. [35] The twisted conjugacy problem has the affirmative 
answer for G being polycyclic-by-finite group and H be equal to a unique 
automorphism (p. 

Proof. It follows immediately from Theorem 2.6 by the same argument 
as in the paper of Mal'cev [69] (see also [75], where the propperty 
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of conjugacy separability was first formulated) for the (non-twisted) 
conjugacy problem. □ 

In fact we have proved the following statement. 

Theorem 2.11. [35] If G is strongly twisted conjugacy separable then 
the twisted Dehn conjugacy problem is solvable for any automorphism 
ofG. 

Corollary 2.12. The twisted conjugacy problem has the affirmative 
answer for G being wreath products A I Z and any automorphism of G, 
where A is a finitely generated abelian group. 

Proof. It is proved in [54] that G is strongly twisted conjugacy separa- 
ble. □ 

Also one can study some more particular cases of this problem. In 
particular, one has 

Theorem 2.13. [35] Let G be a (p-conjugacy separable group. Then 
the twisted Dehn conjugacy problem is solvable for 0. 

From Corollary 3.4 and Proposition 3.5 in [72] one can obtain 

Theorem 2.14. [35] Suppose G is the fundamental group of a closed 
hyperbolic surface and cj) : G ^ G is virtually inner. Then the twisted 
Dehn conjugacy problem is solvable for 0. 

Recently, in [9] the twisted Dehn conjugacy problem was solved 
for virtually surface groups and an example of a finitely presented 
group with solvable conjugacy problem but unsolvable twisted con- 
jugacy problem was given. 

2.7. Some questions. [35] It is evident, that any conjugacy separa- 
ble group is residually finite (because the unity element is an entire 
conjugacy class). This argument does not work for general Reidemeis- 
ter classes. In this relation we have formulated in [35] several open 
questions: 
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Question 1: Does the 0-conjugacy separability imply residually 
finiteness? 

Question 2: Does the 0-conjugacy separability imply residually 
finiteness, provided R{(p) < oo7 

Question 3: Does the twisted conjugacy separability imply residu- 
ally finiteness, provided the existence of (p with R{4>) < oo? 

Question 4: Let G be a residually finite group and its automor- 
phism with R{(j)) < oo. Is G 0-conjugacy separable ? 

The affirmative answer to the last question implies twisted Burnside- 
Frobenius theorem for (p. 



3. Groups with property Rao 
Consider a group extension respecting homomorphism 

(3.1) H ^ G G/H 







H 



G 



G/H 



0, 



where if is a normal subgroup of G. First, notice that the Reidemeister 
classes of in G are mapped epimorphically onto classes oi (j) in G/H. 
Indeed, 



(3.2) 



Suppose that the Reidemeister number -R(0) is infinite, the previous 
remark then implies that the Reidemeister number /?(</)) is infinite. 

Let G be a group, and let ip be an automorphism of G of order m. 
G^ be the group G Z„, = (G, t\ \fg G G, tgt~^ = if{g), = 1). 

The following lemma was proven by Delzant. 



Lemma 3.1. [65, Lemma 3.4] If K is a normal subgroup of a group T 
acting non- elementarily on a hyperbolic space, and ifV/K is Abelian, 
then any coset of K contains infinitely many conjugacy classes. 
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Theorem 3.2. [36] IfG^p has a non- elementary action by isometries on 
a Gromov-hyperbolic length space, then G has infinitely many ip-twisted 
conjugacy classes. 

Proof. By elementary action, we mean an action consisting of ellip- 
tic elements, or with a global fixed point, or a global fixed pair, in 
the boundary of the hyperbolic space. The statement of the theorem 
follows immediately from Lemma 2.1 and Delzant Lemma 3.1 □ 

Theorem 3.3. The following groups have Roo property .■ 

(1) non- elementary Gromov hyperbolic groups [23, 65] 

(2) non- elementary relatively hyperbolic groups, 

(3) the mapping class groups Mods of a compact surface S (with a 
few exceptions) [36], 

(4) the full braid groups Bn{S'^) on n strings of the sphere S"^ [36], 

Proof. (l)-(2): Theorem 3.2 applies if G is a Gromov-hyperbolic group 
or relatively hyperbolic group and if (f has finite order in Out{G). In 
fact, in this case, G^p contains G as a subgroup of finite index, thus 
is quasi-isometric to G, and by quasi-isometry invariance, it is itself 
a Gromov-hyperbolic or relatively hyperbolic group. Now let assume 
that a automorphism of a hyperbolic or relatively hyperbolic group has 
infinite order in Out{G). We describe main steps of the proof in this 
case ( see [23, 65] for the details). By [79] and [5] $ preserves some R- 
tree T with nontrivial minimal small action of G{ recall that an action 
of G is small if all ars stabilisers are virtually cyclic; the action of G 
on T is always irreducible( no global fixed point, no invariant line, no 
invariant end)). This means that there is an i?-tree T equipped with 
an isometric action of G whose length function satisfies / ■ $ = A/ for 
some A > 1. 

Step 1. Suppose A = 1. Then the Reidemeister number R{(l)) is infinite 

Step 2. Suppose A > 1. Assume that arc stabilisers are finite, and 
there exists Nq ^ N such that, for every Q & T, the action of StabQ 
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on TToiT — Q) has at most Nq orbits. Then the Reidemeister number 
-R(0) is infinite. 

Step 3. If A > 1, then T has finite arc stabihsers. If A > 1 then from 
work of Bestwina-Feighn [4] it follows that there exists Nq ^ N such 
that, for every Q E T, the action of StabQ on 7ro(T — (Q)) has at most 
A'o orbits. 

(3) Now let S be an oriented, compact surface of genus g and with p 
boundary components, where 3g + p — A > 0. It is easy to see that the 
mapping Class Group Mods is a normal subgroup of the full mapping 
class group Modg, of index 2. The graph of curves of S, denoted ^(5*), 
is the graph whose vertices are the simple curves of S modulo isotopy. 
Two vertices (that is two isotopy classes of simple curves) are linked by 
an edge in this graph if they can be realized by disjoint curves. Both 
Mods aiid Mod*g act on Q{S) in a non-elementary way. Now we use 
the non-elementary result of Masur and Minsky [70] (see also Bowditch 
[10]) that the complex of curves of an oriented surface (with genus g 
and p boundary components, and 3g+p — 4 > 0) is Gromov-hyperbolic 
space. 

Thus Theorem 3.2 is applicable for Mods aiid for ipi the automor- 
phism induced by reversing the orientation of S, since in this case, 
(Mods)^^ = Mods>i^-^1^2 — Modg. For Mods and (po = Id, we 
have R{(fo = Id) = oo because the group Mods has infinite an num- 
ber of usual conjugacy classes. Finally, Out{Mods) — {ipojTpi} (see 
[57]), which ensures that Mods has the i?oo property if S is an ori- 
entable, compact surface of genus g with p boundary components, 
where 3g + p — A > 0. The only cases not covered by this inequal- 
ity are: i) S is the torus with at most one hole and ii) S is the sphere 
with at most 4 holes. The case of the torus with at most one hole 
follows from the section 3 in [36]. In the case of the sphere with at 
most 4 holes, this follows directly from the knowledge of Out(ModS) 
and the cardinality of the mapping class group. 
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(4) Let : 5„(S'^) Bni^S"^) be an automorphism. Since the center 
of Bn{S^) is a characteristic subgroup, induces a homomorphism of 
the short exact sequence 

1 ^ Z2 ^ Bn{S^) ^ Modsi ^ 1' 

where = S"^ — r open disks. This short exact sequence was obtained 
from the sequence in [6]. The result above imphes that the group 
Mod 32 for n > 3, has the i?oo property. Then the remark about exten- 
sions imphes that the group BniS"^) also has this property. For n < 3 
the groups Bn{S'^) are finite so they do not have the -Roo property. 

□ 

Relatively hyperbolic groups were introduced by Gromov[49] and 
since then various characterizations of relatively hyperbolic groups have 
been obtained. Examples of relatively hyperbolic groups are: 

• the free products of finitely many finitely generated groups are 
hyperbolic relative to the factors. 

• geometrically finite isometry groups of Hadamard manifolds of 
negatively pinched sectional curvature are hyperbolic relative to 
the maximal parabolic subgroups. This includes complete finite 
volume manifolds of negatively pinched sectional curvature. 

• The amalgamation of relatively hyperbolic groups over para- 
bolic subgroups is relatively hyperbolic, when the parabolic sub- 
group is maximal in at least one of the factors [Dah03a, Osi06a] 

• CAT(0)-groups with isolated flats are hyperbolic relative to the 
flat stabilizers. Examples of CAT(0)-groups with isolated fiats 
are listed in [HK05]. 

• Sela's limit groups are hyperbolic relative to non-cyclic maximal 
abelian subgroups [Dah03a] 

3.1. Asymptotic expansions. Suppose we know that the number of 
the twisted conjugacy classes of a automorphism of a group G is 
infinite. The next natural step will be to write an asymptotic for the 
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number of twisted conjugacy classes with a norm smaller then x. First 
of all we need to define a norm of a twisted conjugacy class. In present 
section we realise this approach for pseudo-Anosov homeomorphism of 
a compact surface. 

We assume X to be a compact surface of negative Euler characteristic 
and f : X ^ X is a. pseudo-Anosov homeomorphism, i.e. there is a 
number A > 1 and a pair of transverse measured foliations {F^, fi^) and 
such that = (F^ i/i^) and = (F",A/i"). 

The mapping torus Tf of f : X ^ X is the space obtained from 
X X [0, 1] by identifying (x, 1) with (/(x), 0) for all x E X. It is often 
more convenient to regard Tf as the space obtained from X x [0, oo) 
by identifying (x, s + 1) with (/(x), s) for all x G X, s G [0, oo). On Tj 
there is a natural semi-flow (f) : TjX [0, oo) — > Tf, 4>t{x, s) = (x, s + t) for 
all t > 0. Then the map f : X ^ X is the return map of the semi-flow 
(f). A point X G X and a positive number r > determine the orbit 
curve 4>(x,t) '■= 0i(3;)o<t<T '^'^^ fixed points and periodic points 

of / then correspond to closed orbits of various periods. Take the base 
point Xq of X as the base point of Tf. According to van Kampen's 
Theorem, the fundamental group G := 7ri(Tj,Xo) is obtained from it 
by adding a new generator z and adding the relations z^z~^ = f*{g) 
for all (7 G vr = 7ri(X, Xq), where z is the generator of 7ri(S'"'^, Xq). This 
means that G is a semi-direct product G = tt x Z of tt with Z. 

There is a canonical projection t : Tf ^ R/Z given by (x, s) ^—>- s. 
This induces a map 7ri(r) : G = 7ri(T/-,Xo) Z. 

The Reidemeister number R{f) is equal to the number of homo- 
topy classes of closed paths 7 in Tf whose projections onto R/Z are 
homotopic to the path 

a: [0,1] ^ R/Z 
s \—*s. 

Corresponding to this, there is a group-theoretical interpretation of 
R{f) as the number of usual conjugacy classes of elements 7 G vri(T/-) 
satisfying vri(r)(7) = z. 
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Lemma 3.4. [93, 78] The interior of the mapping torus Int(Tf) admits 
a hyperbolic structure of finite volume if and only if f is isotopic to a 
pseudo-Anosov homeomorphism. 

So, if the surface X is closed and / is isotopic to a pseudo-Anosov 
homeomorphism, the mapping torus Tj can be reahsed as a hyper- 
bohc 3-manifold , H^/G, where is the Poincare upper half space 
{{x,y,z) : z > 0,{x,y) G R^} with the metric ds"^ = {dx"^ + dy"^ + 
dz'^)/z^. The closed geodesies on a hyperbolic manifold are in one- 
to-one correspondence with the free homotopy classes of loops. These 
classes of loops are in one-to-one correspondence with the conjugacy 
classes of loxodromic elements in the fundamental group of the hy- 
perbolic manifold. This correspondence allowed Ch. Epstein (see 
[20], p. 127) to study the asymptotics of such functions as p„(x) = 
^{primitive closed geodesies of length less than x represented by an 
element of the form gz^} using the Selberg trace formula. A prim- 
itive closed geodesic is one which is not an iterate of another closed 
geodesic. Later, Phillips and Sarnak [80] generalised results of Epstein 
and obtained for n-dimensional hyperbolic manifold the asymptotic of 
the number of primitive closed geodesies of length at most x lying in 
fixed homology class. The proof of this result makes routine use of the 
Selberg trace formula. In the more general case of variable negative 
curvature, such asymptotics were obtained by Pollicott and Sharp [81]. 
They used a dynamical approach based on the geodesic flow. We will 
only need an asymptotic for pi{x). Note that closed geodesies repre- 
sented by an element of the form gz are automatically primitive, be- 
cause they wrap exactly once around the mapping torus (once around 
the generator z). We have following asymptotic expansion [20, 80, 81] 

(3.3) pi(x) = -^{Y. + ^(^))' 

n=0 

for any > 0, where h = dim Tf — 1 = 2 is the topological entropy 
of the geodesic flow on the unit-tangent bundle STf, and the constant 
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Co > depends on the volume of hyperbolic 3-manifold Tf. N. Anan- 
tharaman [1] has shown that the constants C„ vanish if n is odd. So, 
we have the following leading asymptotic behaviour: 

(3.4) pi{x) ~ Co^^, asx oo 

Notation. We write f{x) ~ g{x) if — > 1 as x ^ oo. 

Now, using the one-to-one correspondences in Lemma 2.1 and Reide- 
meister bijection in Itroduction we define the norm of the lifting class, 
or of the corresponding twisted conjugacy class {g} in the fundamen- 
tal group of the surface vr = 7ri(X, xq), as the length of the primitive 
closed geodesic 7 on Tj, which is represented by an element of the form 
gz. So, for example, the norm function /* on the set of twisted conju- 
gacy classes equals I* = I o B, where / is length function on geodesies 
(/(7) is the length of the primitive closed geodesic 7) and a norm map 
i? is a bijection defined by formula B{g) = gz between the fundamental 
group of the surface tt = 7ri(X, xq) and the first coset vri(X, xq)z in the 
fundamental group G := 7ri(T/,Xo) = vr xi Z. 

Lemma 3.5. 1) B{^g~f,{{^)~^) = ^B{g)T^ 

2) Function I* = I o B is a non negative twisted class function on G 

Proof. 1) B{-igf\{{-i)-^) = -fgf^{-f-^)z = -/gz^y'^z'^z = -/gz-/'^ = 
lB{g)r^ 

2) /*(7^7/*((7)"') = /o5(7^/,((7)-i) = l{^B{g)T') = l{B{g)) = 
l*{9) □ 

In [27] we introduced the following counting functions: 
Tw(x)= #{twisted conjugacy classes for in the fundamental group 
of surface of norm less than x}, 

L{x)= # {lifting classes of / of norm less than x}. A norm function on 
the set of liftings of / equals /** = I* o L, where /* is defined as above 
and L{f' = a o /) = a is a function from the set of liftings of / to the 
group of their coordinates vr = 7ri(X, xq) (see the introduction). The 
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function /** is a class function on the set of liftings, constant on the 
liftings classes. 

Theorem 3.6. [27] Let X be a closed surface of negative Euler char- 
acteristic and let f : X ^ X he a pseudo-Anosov homeomorphism. 



where the constant Co > depends on the volume of the hyperbolic 
3-manifold Tj, and the constants Cn vanish if n is odd. 

Proof. The proof follows from Reidemeister bijection, Lemma 3.4, 
Lemma 2.1 and the asymptotic expansion (3.1). 

Corollary 3.7. For pseudo-Anosov homeomorphisms of closed sur- 
faces the Reidemeister number is infinite. 

Question 3.1. How can one define the norm of a twisted conjugacy 
class in the general case? 

4. Nielsen fixed point theory and symplectic Floer 



In the dimension two a diffeomorphism is symplectic if it preserves 
area. As a consequence, the symplectic geometry of surfaces lacks 
many of the interesting phenomena which are encountered in higher 
dimensions. For example, two symplectic automorphisms of a closed 
surface are symplectically isotopic iff they are homotopic, by a theo- 
rem of Moser[74]. On other hand symplectic fixed point theory is very 
nontrivial in dimension 2, as it is shown by the Poincare-Birkhoff theo- 
rem. It is known that symplectic Floer homology on surface is a simple 
model for the instanton Floer homology of the mapping torus of the 
surface diffeomorphism [86]. 

4.1. Symplectic Floer homology. 



Then 




n=0 
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4.1.1. Monotonicity. In this section we discuss the notion of mono- 
tonicity as defined in [86, 43]. Monotonicity plays important role for 
Floer homology in two dimensions. Throughout this article, M de- 
notes a closed connected and oriented 2-manifold of genus > 2. Pick 
an everywhere positive two-form u on M. 

Let (p G Symp(M, cj), the group of symplectic automorphisms of the 
two-dimensional symplectic manifold {M,uj). The mapping torus of 0, 
= MxM/(t+l, x) ~ (t, is a 3-manifold fibered over = R/Z. 

There are two natural second cohomology classes on T^, denoted by [u^] 
and c^. The first one is represented by the closed two-form uj^ which 
is induced from the puUback of to M x M. The second is the Euler 
class of the vector bundle V"^ = M x TM/ {t + l,C,x) ~ {t, d^^^^^)? which 
is of rank 2 and inherits an orientation from TM. 

(p G Symp(M, cj) is called monotone, if [uj^ = (areatj(M)/x(M))-C0 
in H'^{T^]'R); throughout this article Symp™'(M, cj) denotes the set of 
monotone symplectomorphisms. 

Now H^(T^;M.) fits into the following short exact sequence [86, 43] 

(4.1) ^ . ^ H\T,;R) ^ H\M;R), 0. 

image(id— 0*) 

where the map r* is restriction to the fiber. The map d is defined 
as follows. Let p : / ^ M be a smooth function which vanishes near 
and 1 and satisfies J^pdi = 1. If 6' is a closed 1-form on M, then 
p-6Adt defines a closed 2-form on T^; indeed d[6] = [p-6Adt]. The map 
r : M ^ assigns to each x G M the equivalence class of (1/2, x). 
Note, that r*uj^ = u and r*c^ is the Euler class of TM. Hence, by 
(4.1), there exists a unique class m(</)) G H^{M; M) / image(id —(f)*) sat- 
isfying dm{(j)) = [uj^] — (areaa;(M)/x(M)) ■ c^, where x denotes the 
Euler characteristic. Therefore, is monotone if and only if m(0) = 0. 

We recall the fundamental properties of Symp'"(M, u;) from [86, 43]. 
Let Diff+(M) denotes the group of orientation preserving diffeomor- 
phisms of M. 
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(Identity) idi/ G Symp'"(M, cj). 

(Naturality) If e Symp™(M, tu), e Diff+(M), then ^'"VV' e 
Symp"*(M,7^*u;). 

(Isotopy) Let {'ipt)t€i be an isotopy in Symp(M, tu), i.e. a smooth 
path with ipo = id. Then 

m(0 o ^/^i) = m{(j)) + [Flux(?/;t)<e7] 

in iy^(M; M)/ image(id — 0*); see [86, Lemma 6]. For the definition of 
the fiux homomorphism see [73]. 

(Inclusion) The inclusion Symp'"(M, cu) > Diff'''(M) is a homotopy 
equivalence. 

(Floer homology) To every (p G Symp'"(M, cj) symplectic Floer ho- 
mology theory assigns a Z2-graded vector space HF^^cf)) over Z2, with 
an additional multiplicative structure, called the quantum cap prod- 
uct, H*{M;Z2) ® HF^{(f)) — ^ HF^{^). For = idu the symplec- 
tic Floer homology i7F*(idA/) are canonically isomorphic to ordinary 
homology iJ*(Af;Z2) and quantum cap product agrees with the or- 
dinary cap product. Each ip G Diff'''(M) induces an isomorphism 
i7F,(0) = HF^{'ip-^(f)'^) of ^r*(M;Z2)-modules. 

(Invariance) If 0,0' G Symp'"(M, cu) are isotopic, then HF^{(f)) and 
HF^:{(f)') are naturally isomorphic as if*(M; Z2)-modules. 

This is proven in [86, Page 7]. Note that every Hamiltonian pertur- 
bation of (see [15]) is also in Symp'"(M, a;). 

Now let g he a. mapping class of M, i.e. an isotopy class of Diff'''(M). 
Pick an area form u and a representative G Symp'^(M, u) of g. Then 
HF^:{(f)) is an invariant of g, which is denoted by HF^{g). Note that 
HF^,{g) is independent of the choice of an area form u by Moser's 
isotopy theorem [74] and naturality of Floer homology. 

4.L2. Floer homology. Let G Symp(M, u;). There are two ways of 
constructing Floer homology detecting its fixed points, Fix(0). Firstly, 
the graph of is a Lagrangian submanifold of M x M, (— cu) x uo) and its 
fixed points correspond to the intersection points of graph(0) with the 
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diagonal A = {{x, x) E M x M}. Thus we have the Floer homology of 
the Lagrangian intersection HF^{M x M, A, graph{(f))). This intersec- 
tion is transversal if the fixed points of are nondegenerate, i.e. if 1 is 
not an eigenvalue of d(j){x), for x G Fix(0). The second approach was 
mentioned by Floer in [38] and presented with details by Dostoglou 
and Salamon in [15] .We follow here Seidel's approach [86] which, com- 
parable with [15], uses a larger class of perturbations, but such that the 
perturbed action form is still cohomologous to the unperturbed. As a 
consequence, the usual invariance of Floer homology under Hamilton- 
ian isotopies is extended to the stronger property stated above. Let 
now (f) G Symp™'(M, cj), i.e (f) is monotone. Firstly, we give the defini- 
tion of HF^{(j)) in the special case where all the fixed points of are 
non-degenerate, i.e. for all y G Fix(()0), det(id— d0j^) 7^ 0, and then 
following Seidels approach [86] we consider general case when has 
degenerate fixed points. Let = {y e C°°(M, M) | y{t) = (f){y{t + l))} 
be the twisted free loop space, which is also the space of sections of 
S^. The action form is the closed one-form on defined 
by a^{y)Y = u{dy /dt,Y{t)) dt, where y ^ Vt^ and Y G Tyil^, i.e. 
Y{t) G Ty(t)M and Y{t) = d(j)y^t+i)Y{t + 1) for all t G M. 

The tangent bundle of any symplectic manifold admits an almost 
complex structure J : TM — > TM which is compatible with u in sense 
that {v,w) = uj{v,Jw) defines a Riemannian metric. Let J = {Jt)tm 
be a smooth path of cu-compatible almost complex structures on M 
such that Jt+i = (f)*Jt. If Y,Y' G TyVl^,, then uj{Y'{t), JtY{t))dt 
defines a metric on the loop space Q^. So the critical points of a^^ are 
the constant paths in and hence the fixed points of 0. The negative 
gradient lines of a^j with respect to the metric above are solutions of 
the partial differential equations with boundary conditions 

r =0(M(s,t+ 1)), 

(4.2) I dsu + Jtiu)dtu = 0, 

[ \ims^±oo u{s,t) G Fix(0) 

These are exactly Gromov's pseudoholomorphic curves [48]. 
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For G Fix(()0), let A4{y~,y'^; J,(f)) denote the space of smooth 
maps n : ^ M which satisfy the equations (4.2). Now to every u G 
Ai{y~,y~^; J,(f)) we associate a Fredholm operator which hnearizes 
(4.2) in suitable Sobolev spaces. The index of this operator is given 
by the so called Maslov index fi{u), which satisfies fi{u) = deg{y~^) — 
deg{y~) mod 2, where (— l)'^'=s?/ = sign(det(id — d</)y)). We have no 
bubbling, since for surface 7i2{M) = 0. 

For a generic J, every u G Ai{y~ ,y^; J,(f)) is regular, meaning that 
D„ is onto. Hence, by the implicit function theorem, Aik{y~ ,y~^; J,(f)) 
is a smooth /c-dimensional manifold, where Aik{y~ ,y~^', J,4') denotes 
the subset of those u G Ai{y~ ,y~^] J,(j)) with fi{u) = A; G Z. Transla- 
tion of the s- variable defines a free M-action on 1-dimensional manifold 
A4i{y~ ,y~^; J.,(j)) and hence the quotient is a discrete set of points. 

The energy of a map n : ^ M is given by 



for all y G Fix(</)). P.Seidel has proved in [86] that if (f) is monotone, 
then the energy is constant on each A4k{y~ ,y~^', J,(p)- Since all fixed 
points of are nondegenerate the set Fix(0) is a finite set and the Z2- 
vector space CF^{(f)) := zf^^^^'^^ admits a Z2-grading with (^—I'^'^^^y = 
sign(det(id — d0j,)), for all y G Fix(</)). The boundness of the en- 
ergy E{u) for monotone (p implies that the 0-dimensional quotients 
M.i{y^,y+, J, 0)/M are actually finite sets. Denoting by n{y_,y^) the 
number of points mod 2 in each of them, one defines a differential 
dj : CF40) ^ CF*+i(0) by djy^ = T.y^n{y.,y+)y+. Due to gluing 
theorem this Floer boundary operator satisfies dj o dj = 0. For gluing 
theorem to hold one needs again the boundness of the energy E{u) . 
It follows that {CF^{(f)), dj) is a chain complex and its homology is by 
definition the Floer homology of denoted HF^[(j)). It is independent 
of J and is an invariant of (p. 

If has degenerate fixed points one needs to perturb equations (4.2) 
in order to define the Floer homology. Equivalently, one could say that 
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the action form needs to be perturbed. The necessary analysis is given 
in [86], it is essentially the same as in the slightly different situations 
considered in [15]. But Seidel's approach also differs from the usual 
one in [15]. He uses a larger class of perturbations, but such that the 
perturbed action form is still cohomologous to the unperturbed. 

4.2. Nielsen numbers and Floer homology. 

4.2.1. Periodic diffeomorphisms. 

Lemma 4.1. [59] Let a non-trivial orientation preserving periodic 
diffeomorphism of a compact connected surface M of Euler character- 
istic x{M) < 0. Then each fixed point class of (p consists of a single 
point. 

There are two criteria for monotonicity which we use later on. Let 
u be an area form on M and G Symp(S, u). 

Lemma 4.2. [43] Assume that every class a G ker(id —0*) C Hi{M; Z) 
is represented by a map 7 : S* — > Fix{(j)), where S is a compact oriented 
1-manifold. Then cf) is monotone. 

Lemma 4.3. [43] If (j)^ is monotone for some k > 0, then is mono- 
tone. If (p is monotone, then 0^ is monotone for all k > 0. 

We shall say that : M — > M is a periodic map of period m, if 0™ 
is the identity map idM : M — > M. 

Theorem 4.4. [37] If (p is a non-trivial orientation preserving periodic 
diffeomorphism of a compact connected surface M of Euler character- 
istic x{^) < Oj then is monotone with respect to some cp-invariant 
area form and 

HF,{(P) = Z^''^\ dim HF,{(P) = N{(P), 



where N{(p) denotes the Nielsen number of (p. 
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Proof. Let be a periodic diffeomorphism of least period /. First note 
that if u) is an area form on M, then area form u := X]i=i(<i'^*)*'^ is 0- 
invariant, i.e. G Symp(M, cu). By periodicity of 0, 0' is the identity 
map idM : M — ^ M. Then from Lemmas 4.2 and 4.3 it follows that u 
can be chosen such that G Symp'"(M, cu). 

Lemma 4.1 implies that every y G Fix(0) forms a different fixed 
point class of 0, so #Fix(0) = N{(f)). This has an immediate conse- 
quence for the Floer complex {C F^{(j)) , dj) with respect to a generic 
J = {Jt)t&R. If G Fix(0) are in different fixed point classes, then 
J, 0) = 0. This follows from the first equation in (4.2). 
Then the boundary map in the Floer complex is zero dj = and Z2- 
vector space CF^{(j)) := zf^^^^'^'^ = ll^'^'^^ . This immediately implies 
i7F,(0) = lli^'^^ and dimi7F,(0) = A^(0). 

□ 

4.2.2. Algebraically finite mapping classes. A mapping class of M is 
called algebraically finite if it does not have any pseudo-Anosov compo- 
nents in the sense of Thurston's theory of surface diffeomorphism. The 
term algebraically finite goes back to J. Nielsen. 

In [43] the diffeomorphisms of finite type were defined . These are spe- 
cial representatives of algebraically finite mapping classes adopted to 
the symplectic geometry. 

Definition 4.1. [43] We call G Diff+(M) of finite type if the fol- 
lowing holds. There is a 0-invariant finite union C M of disjoint 
non-contractible annuli such that: 

(1) 0|M\ is periodic, i.e. there exists £ > such that 0^|M\ A^ = id. 

(2) Let A^' be a connected component of A^ and £' > be the smallest in- 
teger such that 0^' maps A^' to itself. Then 0^' | A^' is given by one of the 
following two models with respect to some coordinates {q,p) G / x S"^: 



(twist map) 



{q,p) I — ^ {q,p- fiq)) 
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(flip-twist map) (g,p) i — ^ (1 - q, -p- f{q)), 



where / : J ^ M is smooth and strictly monotone. A twist map is 
called positive or negative, if / is increasing or decreasing. 

(3) Let A^' and £' be as in (2). If ^' = 1 and (j)\N' is a twist map, then 
image(/) C [0, 1], i.e. 0|int(A^') has no fixed points. 

(4) If two connected components of are homotopic, then the corre- 
sponding local models of (p are either both positive or both negative 
twists. 

The term fiip-twist map is taken from [61]. 

By Mid we denote the union of the components of M\int(A^), where 
restricts to the identity. 

The next lemma describes the set of fixed point classes of (p. It is a 
special case of a theorem by B. Jiang and J. Guo [61], which gives for 
any mapping class a representative that realizes its Nielsen number. 

Lemma 4.5 (Fixed point classes). [61] Each fixed point class of (p is 
either a connected component of Mid or consists of a single fixed point. 
A fixed point x of the second type satisfies det(id— d0j;) > 0. 

The monotonicity of diffeomorphisms of finite type was investigated 
in details in [43]. Let be a diffeomorphism of finite type and I be 
as in (1). Then 0^ is the product of (multiple) Dehn twists along A^. 
Moreover, two parallel Dehn twists have the same sign, by (4). We say 
that has uniform twists, if 0^ is the product of only positive, or only 
negative Dehn twists. 

Furthermore, we denote by £ the smallest positive integer such that 0^ 
restricts to the identity on M \ A^. 

If uj' is an area form on M which is the standard form dg A dp 
with respect to the (g,p)-coordinates on A^, then u := Yli=ii^^)*^' 
standard on A^ and 0-invariant, i.e. G Symp(M, cj). To prove that 
UJ can be chosen such that G Symp'"(M, lu), Gautschi distinguishes 
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two cases: uniform and non-uniform twists. In the first case he proves 
the following stronger statement. 

Lemma 4.6. [43] // has uniform twists and u is a (p-invariant area 
form, then (p G Symp™(M, cj). 

In the non-uniform case, monotonicity does not hold for arbitrary 
0-invariant area forms. 

Lemma 4.7. [43] If (p does not have uniform twists, there exists a 0- 
invariant area form uj such that (p G Symp"*(M, u;). Moreover, uj can 
he chosen such that it is the standard form dg A dp on N . 

Theorem 4.8. [37] If cp is a diffeomorphism of finite type of a compact 
connected surface M of Euler characteristic x{^) < and if (p has 
only isolated fixed points , then cp is monotone with respect to some 
(p-invariant area form and 

HFM = dim HFM = iV(0), 

where N{(p) denotes the Nielsen number of (p. 

Proof. From Lemmas 4.6 and 4.7 it follows that uj can be chosen such 
that (p G Symp'"(M, u;). Lemma 4.5 implies that every y G Fix(0) 
forms a different fixed point class of </>, so #Fix(</)) = N{(p). This has 
an immediate consequence for the Floer complex {CF^{(p),dj) with 
respect to a generic J = {Jt)teM.- If Z/^ ^ Fix(0) are in different fixed 
point classes, then Ai{y~ ,y~^; J,(p) = 0. This follows from the first 
equation in (4.2). Then the boundary map in the Floer complex is 
zero dj = and Z2-vector space CF^{(p) := zf^^^^^^ = tJ^^^^ This 
immediately implies HF^{(p) ^ Z^^^'^ and dimi7F,(0) = N{(p). 

□ 

Theorem 4.9. [43] Let (p he a diffeomorphism of finite type, then cp is 
monotone with respect to some (p-invariant area form and 

Here, L denotes the Lefschetz number. 



36 



ALEXANDER FEL'SHTYN 



Proof. The main idea of the proof is a separation mechanism for Floer 
connecting orbits. Together with the topological separation of fixed 
points discussed in theorem 4.8 , it allows us to compute the Floer 
homology of diffeomorphisms of finite type. 

There exists a function H : M W such that H\mt{Mid) is a Morse 
function, meaning that all the critical points are non-degenerate and 
H\M \ Mid = 0. Let {tpt)tm denote the Hamiltonian flow generated 
by H with respect to the fixed area form u and set $ := </> o ?/;]^. Then 
Fix($) = (crit(if) n Mi^) U ( Fix(0) \ Mid). In particular, $ only has 
non-degenerate fixed points. Let Nq C Mid be a collar neighborhood of 
dMid- Let x~,x^ G Fix($) flMj^ be in the same connected component 
of Mid. If M G A4{x~,x^; J, $), then imagew C S^, where denotes 
the 5-neighborhood of Mid \ Nq with respect to any of the metrics 
ui.^Jt.) [86, 43] 

Moreover, lemma 4.5 implies that every y G Fix(0) \ Mid forms a 
different fixed point class of $. This has an immediate consequence for 
the Floer complex (CF*(<l>),9j) with respect to a generic J = {Jt)tm- 
Namely, (CF*($), dj) splits into the subcomplexes (Ci, di) and (C2, ^2), 
where Ci is generated by cnt{H)r\Mid and C2 by Fix{(f))\Mid. Moreover, 
C2 is graded by and ^2 = [43]. The homology of (Ci,9i) is isomor- 
phic to H,{M,d, d+M,d] Z2) [86, 43]. So HF,{(P) = H,{Mid, Z2) © 
^#Fix(<^)|M\M,d^ gj^pg every fixed point of (j)\M\Mid has fixed point in- 
dex 1, the Lefschetz fixed point formula implies that 7^(Fix(0) \Mjf;) = 
X{4>\M\Mid). 

□ 

Remark 4.2. In the theorem 4.8 the set Mid is empty and every fixed 
point of (f) has fixed point index 1 [61]. The Lefschetz fixed point 
formula implies that #Fix(</)) = N{(f)) = L{(f)) . So, theorem 4.8 
follows also from theorem 4.9. 

4.3. Symplectic zeta functions and asymptotic invariant. 
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4.3.1. Symplectic zeta functions. Let F = 7ro{Dif f~^{M)) be the map- 
ping class group of a closed connected oriented surface M of genus > 2. 
Pick an everywhere positive two-form u on M. A isotopy theorem of 
Moser [74] says that each mapping class of G F, i.e. an isotopy 
class of Dif f~^{M), admits representatives which preserve u. Due to 
Seidel[86] we can pick a monotone representative (p ^ Symp"*(M, cj) 
of g. Then HF^,{(j)) is an invariant of g, which is denoted by HF^{g). 
Note that HF^,{g) is independent of the choice of an area form uj by 
Moser's theorem and naturality of Floer homology. By lemma 4.3 
symplectomorphisms are also monotone for all n > 0. Taking a 
dynamical point of view, we consider the iterates of monotone sym- 
plectomorphism and define the first symplectic zeta function of 
[37] as the following power series: X<t>{'^) = ^xp ^Xl^i ^^^^^^^^-^2;"j , 
where x(ifF^.(0")) is the Euler characteristic of Floer homology group 
of 0". Then Xif){^) is an invariant of g, which we denote by Xgi^)- Let 
us consider the Lefschetz zeta function L^{z) := exp ^^t ^ 



where L{r) := EtT(-l)'tr : Hk{M; Q) ^ Hk{M; Q) 
Lefschetz number of 0". 



is the 



Theorem 4.10. [37] Symplectic zeta function x<i){^) ^■^ o rational func- 
tion of z and 



dimX 

,(-i)'=+i 



X4>{z) = L^{z) = Y[ det (j - 0*fc.z)' 



fc=0 

Proof. If for every n all the fixed points of (p^ are non-degenerate, i.e. 
for all X G Fix(()</)'^), det(id — d0"(x)) ^ 0, then we have( see section 
4): x{HF,{r)) = E..=<^"(x)Sign(det(id-d0"(x))) = If we have 

degenerate fixed points one needs to perturb equations (4.2) in order 
to define the Floer homology. The necessary analysis is given in [86] is 
essentially the same as in the slightly different situations considered in 
[15], where the above connection between the Euler characteristic and 
the Lefschetz number was firstly established. □ 
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In [37] we have defined the second symplectic zeta function for mono- 
tone symplectomorphism as the following power series: F^{z) = 
exp ^Xl^i '^'™^r*'''^ '* ^") ■ Then -^0(2;) is an invariant of mapping class 
g, which we denote by Fg{z). 

Motivation for this definition was the theorem 4.4 and nice analytical 
properties of the Nielsen zeta function N^{z) = exp ^Xl^i '' -^"^^ 
see [22]. We denote the numbers dim if F,, (</)"■) by Nn- Let d e N, 
be the Mobius function. 

Theorem 4.11. [37] Let (p be a non-trivial orientation preserving pe- 
riodic diffeomorphism of least period m of a compact connected surface 
M of Euler characteristic x{^) < . Then the zeta function F^{z) 
is a radical of a rational function and Fff^^z) = Yld\m ~ z'^)~'^^'^\ 
where the product is taken over all divisors d of the period m, and P{d) 
is the integer P{d) = 

Remark 4.3. Given a symplectomorphism of surface M, one can 
form the symplectic mapping torus = xi S^, where T| is usual 
mapping torus (see section 3.1). lonel and Parker [53] have computed 
the degree zero Gromov invariants (these are built from the invariants 
of Ruan and Tian) of and of fiber sums of the with other sym- 
plectic manifolds. This is done by expressing the Gromov invariants 
in terms of the Lefschetz zeta function L^{z) . The result is a large 
set of interesting non-Kahler symplectic manifolds with computational 
ways of distinguishing them. In dimension four this gives a symplectic 
construction of the exotic elliptic surfaces of Fintushel and Stern. In 
higher dimensions it gives many examples of manifolds which are dif- 
feomorphic but not equivalent as symplectic manifolds. Theorem 4.10 
implies that the Gromov invariants of are related to symplectic 
Floer homology of via zeta function X4>{z) = L^{z). We hope that 
the second symplectic zeta function F^{z) give rise to a new invariant 
of symplectic 4-manifolds. 
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4.3.2. Topological entropy and the Nielsen numbers. A basic relation 
between Nielsen numbers and topological entropy h{f) was found by 
N. Ivanov [56]. We present here a very short proof of Jiang of the 
Ivanov's inequality. 

Lemma 4.12. [56] 

h(f) > limsup- -logA^fD 

n n 

Proof Let 6 be such that every loop in X of diameter < 26 is con- 
tractible. Let e > be a smaller number such that d{f{x), f{y)) < 6 
whenever d{x, y) < 2e. Let En C X he a. set consisting of one point 
from each essential fixed point class of j"^. Thus \ En \= A^(/"). By 
the definition of h{f), it suffices to show that En is (n, e)-separated. 
Suppose it is not so. Then there would be two points x ^ y E En such 
that d{f\x), f^{y)) < e for a < i < n hence for all i > 0. Pick a path 
from to /*(?/) of diameter < 2e ioi a < i < n and let = Cq. By 

the choice of 6 and e , / o q ~ Cj+i for all i, so /" o cq — c„ = Cq. such 
that This means x, y in the same fixed point class of /", contradicting 
the construction of En- 

This inequality is remarkable in that it does not require smoothness 
of the map and provides a common lower bound for the topological 
entropy of all maps in a homotopy class. 

We recall Thurston classification theorem for homeomorphisms of 
surfase M of genus > 2. 

Theorem 4.13. [94] Every homeomorphism (p : M ^ M is isotopic 
to a homeomorphism f such that either 

(1) f is a periodic map; or 

(2) f is a pseudo-Anosov map, i.e. there is a number A > 1 (stretching 
factor) and a pair of transverse measured foliations [F^, fi^) and (F", 
such that f{F',fi') = {F', \fi') and f{F'',fi'') = [F'^.X^i^); or 

(3) f is reducible map, i.e. there is a system of disjoint simple closed 
curves 7 = {71, ,7^} in intM such that 7 is invariant by f (but 7^ 
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may be permuted) and 7 has a f -invariant tubular neighborhood U such 
that each component of M\U has negative Euler characteristic and on 
each(not necessarily connected) f -component of M\U, f satisfies (1) 
or (2). 

The map / above is called the Thurston canonical form of /. In (3) 
it can be chosen so that some iterate is a generalised Dehn twist on 
U . Such a / , as well as the / in (1) or (2), will be called standard. A 
key observation is that if / is standard, so are all iterates of /. 

Lemma 4.14. [21] Let f be a pseudo-Anosov homeomorphism with 
stretching factor \ > 1 of surfase M of genus > 2. Then 

M/) = %(A) = limsup-- log iV(r) 

Lemma 4.15. [60] Suppose f is a standard homeomorphism of surfase 
M of genus > 2 and A is the largest stretching factor of the pseudo- 
Anosov pieces( X = 1 if there is no pseudo-Anosov piece). Then 

h{f) = log{\) = hmsup - ■ log iV(r) 
n n 

4.3.3. Asymptotic invariant. The growth rate of a sequence a„ of com- 
plex numbers is defined by 

Growth(a„) := max{l,limsup |a„|"^''"} 

n— »oo 

which could be infinity. Note that Growth(a,„) > 1 even if all a„ = 0. 
When Growth(an) > 1, we say that the sequence a„ grows exponen- 
tially. 

Definition 4.4. We define the asymptotic invariant F°°{g) of mapping 
class (7 G r = 7TQ{Dif f~^{M)) to be the growth rate of the sequence 
{an = dimiirF*(0")} for a monotone representative G Symp'"(M, u;) 
of g: 

:= Growth(dimi7F,(0")) 
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Example 4.5. If is a non-trivial orientation preserving periodic dif- 
feomorphism of a compact connected surface M of Euler characteristic 

< , then the periodicity of the sequence dimifF*(0") implies 
that for the corresponding mapping class g the asymptotic invariant 

F°^{g) := Growth(dimi7F,(0")) = 1 

Example 4.6. Let be a monotone diffeomorphism of finite type 
of a compact connected surface M of Euler characteristic x{^) < 
and g a corresponding algebraically finite mapping class. Let U be 

the open regular neighborhood of the k reducing curves 71, ,7^ in 

the Thurston theorem, and Mj be the component of M \ f/.Let F 
be a fixed point class of 0. Observe from [61] that if F C Mj, then 
ind(F, 0) = ind(F, 0j).So if F is counted in A^(0) but not counted in 
N{(f)j) , it must intersect U. But we see from [61] that a component 
of U can intersect at most 2 essential fixed point classes of 0. Hence 
we have A^(0) < X]j^(0i)-For the monotone diffeomorphism of finite 
type maps (f>j are periodic. Applying last inequality to 0" and using 
remark 4.9 we have 

< dim HF,{r) = diYaH,{M'>2\dM[2^;'L2) + Ar(0"|M \ m/;^) < 

< dimi7,(Mif \ aM/;); Z2) + iV(0'^) 

< dimi7,(Mif \ 9M/;^; Z2) + ^ iV((0)^") + 2k< Const 

j 

by periodicity of 0j. Taking the growth rate in n, we get that asymp- 
totic invariant F°°{g) = 1. 

4.4. Generalised Arnold conjecture. Concluding remarks . Let 

: M — s> M be a Hamiltonian symplectomorphism of a compact sym- 
plectic manifold {M,uj). In the nondegenerate case the Arnold conjec- 
ture asserts that 

2n 

i^Fix{(f)) > dimH,{M,Q) = J2^k{M), 

k=0 
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where 2n = dimM, 6fc(M) = dimi^fc(M, Q). 

The Arnold conjecture was first proved by Ehashberg [19] for Rie- 
mann surfaces. For tori of arbitrary dimension the Arnold conjecture 
was proved in the celebrated paper by Conley and Zehnder[ll]. The 
most important breakthrough was Floer's proof of the Arnold conjec- 
ture for monotone symplectic manifolds [39]. His proof was based on 
Floer homology. His method has been pushed through by Fukaya- 
Ono[42], Liu-Tian[66] and Hofer-Salamon[52] to establish the nonde- 
generate case of the Arnold conjecture for all symplectic manifolds. 

The Hamiltonian symplectomorphism is isotopic to identity map 
idM- In this case all fixed points </> are in the same Nielsen fixed point 
class. The Nielsen number of is or 1 depending on Lefschetz number 
is or not. So, the Nielsen number is very weak invariant to estimate 
the number of fixed points of </> for Hamiltonian symplectomorphism. 
From another side, as we saw in theorem 4.4, for the nontrivial periodic 
symplectomorphism of a surface, the Nilsen number of (p gives an 
exact estimation from below for the number of nondegenerate fixed 
points of 0. These considerations lead us to the following question 

Question 4.7. How to estimate the number of nondegenerate fixed 
points of general(not necessary Hamiltonian) symplectomorphism? 

4.4.1. Algebraically finite mapping class. If -0 is a diffeomorphism of 
finite type of surface M then ip G Symp'^(M, cu) for some -^-invariant 
form u. Suppose that symplectomorphism (p has only non-degenerate 
fixed points and is Hamiltonian isotopic to ip. Then G Symp"*(M, 
and HF^,{(j)) is isomorphic to HF^^i^p). So, from theorem 4.9 it follows 
that 

#Fix(0) > dimi7F*(0) = dimi/F*(^) = 

= dimi7,(M^=id, aM^=id; Z2) + Ar(V;|M \ M^=id) = 
2 

= ^ 6fc(M^=id, 9M^=id; Z2) + N{'iP\M \ M^=id) 

k=0 
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This estimation can be considered as a generalisation of Arnold conjec- 
ture becouse it implies Arnold conjecture for = id. If ip is nontrivial 
orientation preserving periodic diffeomorphism then theorem 4.4 im- 
plies an estimation 

> dim HF,{(j)) = dim HF,{tlj) = iV(?/^) 

This estimation can be considered as a generalisation of Arnold con- 
jecture for a nontrivial periodic mapping class. 

4.4.2. Pseudo-Anosov mapping class. For pseudo-Anosov "diffeomor- 
phism" in given pseudo-Anosov mapping class we also have, as in theo- 
rems 4.4,4.8 and 4.9, a topological separation of fixed points [94, 61, 55], 
i.e the Nielsen number of pseudo-Anosov "diffeomorphism" equals to 
the number of fixed points and there are no connecting orbits between 
them. But we have the following difficulties. Firstly, a pseudo-Anosov 
"diffeomorphism" is a smooth and a symplectic automorphism only on 
the complement of his fixed points set. Nevertheless, M. Gerber and A. 
Katok [45] have found a smooth model for pseudo-Anosov "diffeomor- 
phism" with the same dynamical properties. More precise they have 
constructed for every pseudo-Anosov "diffeomorphism" / a diffeomor- 
phism /' which is topologically conjugate to / through a homeomor- 
phism isotopic to identity. Diffeomorphism /' is a symplectomorphism, 
it has the same fixed points as /, which are also topologically sepa- 
rated, and it has the same Nielsen number as /. Secondly, in the case 
of a pseudo-Anosov "diffeomorphism" and it smooth model, we have 
to deal with fixed points of index —p where p > 1. Such fixed points 
are degenerate from symplectic point of view and therefore need a local 
perturbation. 

If is monotone symplectomorphism with nondegenerate fixed points 
in given pseudo-Anosov mapping class {</>} = then 

> dimi7F,(0) = dim HF^{g) 
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To formulate the generalised Arnold conjecture in this case we need 
to know how to calculate in classical terms the Floer homology for 
monotone symplectomorphism with nondegenerate fixed points which 
represents given pseudo-Anosov mapping class g. For this we need, 
for example, to understand the contribution of degenerate fixed points 
of a smooth model of a pseudo-Anosov "diffeomorphism" to the Floer 
homology. In [18] Floer homology were calculated for certain class of 
pseudo-Anosov maps which are compositions of positive and negative 
Dehn twists along loops in M forming a tree-pattern. 

4.4.3. Reducible mapping class. Generalised Arnold conjecture. Sup- 
pose now that symplectomorphism (p has only non-degenerate fixed 
points and that is Hamiltonian isotopic to a monotone symplectomor- 
phism ip in a. reducible mapping class g which contains pseudo-Anosov 
components gpA_,i = 1, see theorem 4.13). 

Conjecture 4.16. The Floer complex {CF,,{%l))^dj) with respect to 
a generic J = {Jt)tm splits into the subcomplexes (C/m, dfin) and 
{Cpj^,dpA),i = l,...s, where Cfm corresponds to the finite type com- 
ponent gjin of g and C*^, i = 1, ...s correspond to pseudo-Anosov com- 
ponents g^A, i = 1, ....s of g. The homology of {Cfm, dfin) is isomorphic 
to HF,{gfm) = H,{M^=id, 9M^=id; Z2) ©zf '^''''^*''"="^ by theorem 4.8 
and 

#Fix{(j)) > dim HF^{(j)) = dim HF^iifj) = dim HF^{g) = 

s 

= dim HF,{gfm) + ^ dimi7F,(^;^) = 

i=l 

s 

= dimi7,(M^=id,aM^=id;Z2) + A^(V^|M\M^=id) + Xld™^^*(4A) = 

i=l 

2 s 

= h{M^=id, 5M^=id; Z2) + N{^\M \ M^=id) + Yl dimHF^ig;^) 

k=0 i=l 
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4.4.4. Concluding remarks. Due to P. Seidel [87] dim HF^,{(f)) is a new 
symplectic invariant of a four-dimensional symplectic manifold with 
nonzero first Betti number. This 4-manifold produced from symplec- 
tomorphism by a surgery construction which is a variation of ear- 
lier constructions due to McMuUen-Taubes, Fintushel-Stern and J. 
Smith. We hope that our asymptotic invariant and symplectic zeta func- 
tion F^{z) also give rise to a new invariants of contact 3- manifolds and 
symplectic 4-manifolds. 
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